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@ A graphics editor generates an appearance 
construction graph (ACG) which represents an 
image formation process as a directed acydic 
graph (DAG). The ACG includes nodes repre- 
senting image fragments (sources), image 
transformation operations (transforms), and 
outputs (viewable images - sinks) which can be 
displayed, for example, in a window of a display 
screen, and manipulated to construct and edit 
complex images. Accordingly, the displayed 
ACG is a user interface that permits complex 
graphics images to be constructed in a 
straightforward manner, and thus easily under- 
stood and manipulated (edited) by an operator. 
The nodes in the ACG are selectively intercon- 
nected by links (edges) therebetween. The com- 
plex image represented by the ACG can be 
constructed and displayed in a viewer window 
on the display screen by "running" the ACG. 
The graphics editor "runs" the ACG by per- 
forming operations on image fragments as de- 
fined by the interconnected nodes of the ACG. 
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The present invention relates to graphics editors, 
and more particularly to graphics editors for con- 
structing and editing complex graphics images, and 
to methods for forming and editing complex graphics 
images through the use of a user interface. 

The standard What-You-See-ls-What-You-Get 
(WYSIWYG) point-and-click editing paradigm pres- 
ents difficulties for the construction of complex 
graphics images. If the user can view only the final 
WYSIWYG image, it is hard to see and understand its 
composition, that is, its components and their rela- 
tions. Consequently, it is difficult to point to the com- 
ponents and modify them. It is also difficult to reuse 
the same image, whether transformed or modified, 
without interactively copying it one or more times. 

For example, to operate on a graphics Image, 
users need to indicate - that is, refer to - the part or 
parts of the image they wish to affect This is often 
done by pointing with a mouse. Experienced users 
know that when manipulating complex images, it can 
be difficult to find and point to the relevant parts, es- 
pecially when there are dense clusters of overlapping 
figures. Related to this, users sometimes find that 
parts of the image they can see are not recognized 
by the editor, so cannot be pointed at and operated on 
independently. A simple example is when a rectangle 
is created using a "rectangle" tool. It typically is not 
possible to select one of the sides of the rectangle 
and move it away. To do this, the rectangle must be 
created as four line segments. There is no ability to re- 
parse or reregister the rectangle - to decompose it 
into parts other than the ones out of which it was first 
constructed. This ability to reregister is closely related 
to the problem of reference, since, the registration of 
an image determines the parts that can be referred 
to. 

Another problem which arises with the WYSI- 
WYG display screen is the inability of an operator to 
determine the graphical domain in which a portion of 
a complex image is located. Anumberof commercially 
available text/graphics editors are capable of con- 
structing images in both the bitmap and vector do- 
mains. For example, Superpaint® is a graphics editor 
which incorporates some painting (bitmap) and draw- 
ing (vector) capabilities. However, unless the user of 
the system knows (based on experience) in which do- 
main a particular portion of an image is created, the 
user may attempt to perform editing operations on 
that image portion while in a domain in which that 
editing cannot be performed. 

Accordingly, a need exists for a graphics editor 
which conveys the underlying components and oper- 
ations which are performed to produce a complex im- 
age to the operator, and which permits the operator 
to construct and edit complex images easily. 

US-A-4,901,063 discloses an image processing 
apparatus which displays both a sample image as it 
appears before a selected image process is executed. 



and a processed sample image corresponding to an 
image resulting from the selected image process. 
With the apparatus, an operator can obtain process- 
ed sample images and can perform image processing 
5 on an original image by inputting a processing com- 
mand. 

US-A-4,611,306 discloses a word processing 
system having a display menu/chart key for toggling 
back and forth between a graphics image and a menu 
10 display used to define a graphics image. The system 
provides an interface which allows an operator to 
check and revise the definition of a graphics image 
visually. By using the menu/chart key, the operator 
can see how changing a menu item will cause a 
15 graphics image to be changed. When the key is 
pressed while a graphic image is being displayed, a 
most recently processed menu is again displayed. 
This editing system does not permit the viewing of an 
entire complex image formation process, or the sl- 
20 multaneous viewing of an image formation process 
and the image formed thereby. 

US-A-4,928,247 discloses a host system which 
executes one or more application programs which re- 
sult in graphics data structures. These graphics data 
25 structures are then continuously and asynchronously 
traversed. In particular, hierarchical data structures 
are built in memory to permit the definition of complex 
data structures representative of the primitives, attri- 
butes and geometric transformations of three dimen- 
30 sional objects. This data structure is then asynchro- 
nously traversed by a structure walker. 

US-A-4,021,976 discloses a method and system 
for generating dynamic interactive visual representa- 
tions of complex information structures within a com- 
35 puter. Boundaries of the information system are es- 
tablished, and a set of mathematical relationships is 
provided which indicates a degree of correlation be- 
tween parameters of interest to a user and segments 
of information contained within the boundaries. Avis- 
40 ual display is generated for the user which has a plur- 
ality of iconic representations and features corre- 
sponding to the parameters defined by the mathe- 
matical relationships. The iconic representations and 
visual features of the visual display change with 
45 movement of the mathematical relationships within 
the boundaries of the information system according 
to the degree of correlation between parameters of in- 
terest and a segment of information through which 
the mathematical relationships are passing. The 
50 shapes of the icons can be used to denote the type 
of data represented by the icons. 

US-A-4.860,204 discloses a method for con- 
structing computer programs through use of visual 
graphical representations. Computer programs are il- 
55 lustrated as visual roadmaps of an intended sequence 
of actions. Each operational entity of a program graph 
displayed on a screen is represented as an elemental 
unit. The elemental unit is a multidimensional graph- 
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ical unit of programming information which has four 
modes, or "layers" of operation. The elemental units 
reside in very general form in a work station library 
and are optimized by specialization. The system per- 
mits programs to be written using icons instead of by s 
writing source code. 

US-A-4,953,106 discloses a computer Imple- 
mented technique for drawing directed graphs provid- 
ing reduced crossing and improved picture quality. 

It is an object of the present invention to provide io 
a graphics editor which permits complex images to be 
formed and edited easily, and in a straightforward 
manner. 

In accordance with the present invitation, a 
graphics editor generates an appearance construe- is 
tion graph (ACG) which represents a complex image 
formation process as a directed acyclic graph (DAG). 
The ACG includes nodes representing image frag- 
ments (sources), image transformation operations 
(transforms), and outputs (viewable images - sinks) 20 
which can be displayed, for example, in a window of 
a display screen, and manipulated by a user to con- 
struct and edit complex images. Accordingly, the dis- 
played ACG is a user interface that permits complex 
graphics images to be constructed in a straightfor- 25 
ward manner, and thus easily understood and ma- 
nipulated (edited) by an operator. The nodes in the 
ACG are selectively interconnected by user defined 
links (edges) therebetween. The complex image rep- 
resented by the ACG can be constructed and dis- so 
played in a viewer window on the display screen by 
"running" the ACG. The graphics editor "runs" the 
ACG by performing operations on image fragments 
as defined by the interconnected nodes of the ACG. 

An operator constructs an ACG graphically in a 35 
graph view window by selecting node types, placing 
icons for these types in the graph view window, and 
creating links (edges) between the nodes. The oper- 
ator can add or delete nodes or links at any time. Ad- 
ditionally, the image fragments (source node con- 40 
tents) can be edited or imported by, for example, ref- 
erence to a library of source images. When editing inv 
age fragments, another window (I.e., an editor win- 
dow) is displayed which permits the operator to edit 
the contents of a selected source node. Once an ACG 45 
has been constructed, and content created or import- 
ed for each source node, the ACG can be run and its 
results displayed. 

The present invention is implemented by directly 
representing ACGs in an object-oriented graph repre- so 
sentation language. The nodes in the graph are ob- 
jects which keep track of their connections to neigh- 
bors and which respond to a standard set of graph tra- 
versal messages. The graphics functionality of nodes 
is implemented by making their objects inherit func- 55 
tionality not only from the node's dass (type of node) 
but also from a graphics operator dass which re- 
sponds to standard operation messages such as 



reading inputs and providing outputs. Node types cor- 
respond to subdasses of the generic operator dass 
which implement functionality appropriate to the node 
type. 

As an operator uses the graph-construction in- 
terface of the present invention to build an ACG, the 
editor builds the ACG in the underlying object model. 
When the operator requests that the ACG be execut- 
ed, the editor runs a graph traversal algorithm which, 
as it walks the graph (e.g., from leaf nodes toward root 
nodes), invokes the graphics operations associated 
with each source, transform, and sink. This has the 
effect of constructing and displaying the appropriate 
appearance (complex output image). That is, the edi- 
tor executes ACGs by propagating graphics informa- 
tion along the dataflow arcs which connect the nodes 
of the ACGs. 

Thus, the present invention permits an operator 
to control complex image formation by manipulating 
a displayed graph of the image formation process. An 
operator is able to develop an image formation proc- 
ess by forming and modifying a graphical display of 
the image formation process. 

The invention will be described by way of exam- 
ple with reference to the following drawings in'which 
like reference numerals refer to like elements and 
wherein: 

Figure 1 is a block diagram showing general com- 
ponents that can be used with the present inven- 
tion; 

Figure 2 is a view of a display screen having an 
ACG in one window and a constructed output im- 
age in another window: 

Figure 3 is a view of a display screen having win- 
dows containing different types of source image 
editors; 

Figure 4 is a view of a display screen having an 
ACG of Figure 2 modified and the Viewer win- 
dows which result; 

Figure 5 is a view of a display screen resulting 
from a further modification of the Figure 4 ACG; 
Figure 6 is a view of a display screen resulting 
from a modification of the Figure 5 ACG; 
Figure 7 is a view of a display screen illustrating 
the modification of a polygon defined by Eudi- 
dean line segments; 

Figure 8 is a view of a display screen containing 

a second ACG and Viewer windows as well as a 

Prompt Window; 

Figure 9 is a list of node icons; 

Figures lOAand 10B are ACGs of source nodes 

attached to sink nodes; 

Figure 11 is an ACG of a Euclidean line segment 
source node attached to a Viewer node by a scan- 
conversion transform; 

Figure 12A is an ACG of a parameterized scan- 
conversion node attached to a Viewer node; 
Figure 12B is an ACG of a dipping transform at- 
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tached to a Viewer node; 

Figure 13 is an enlarged view of the Figure 8 

ACG; 

Figure 14 is a block diagram of the nnodules form- 
ing the graphics editor of the present invention; 
Figure 15 is a flow diagram of an image formation 
process according to the present invention; 
Figure 16 is a flow diagram of the ACG construc- 
tion process of the present invention; 
Figure 1 7 is a flow diagram of the ACG execution 
process of the present invention; and 
Figures 18A and 18B are flow diagrams of ACG 
editing processes. 

The present invention relates a graphics editor, 
which produces a desired graphics image (or appear- 
ance). The edition manipulates image data drawn 
from more than one f igural model (graphical domain). 
A figural model as defined herein is what is referred 
to in graphics literatui-e as an imaging model - a spec- 
ification of constructable appearances in terms of the 
operations available for marking a surface. Specifi- 
cally, the present invention can manipulate both ras- 
ter graphics apparatus of (bitmap graphics), which 
are figures defined over a discrete, cellular grid, and 
line-segment graphics (vector graphics) which are fig- 
ures constructed using line segments in the real 
plane (where real numbers are approximated by com- 
puter floating point numbers). The apparatus can also 
be extended to include other figural models, such as, 
for example, text video and three- dimensional figu- 
ral models. 

In the apparatus of the present invention, graph- 
ics images are constructed by directly manipulating 
underlying image fragments and by applying graphics 
operators (transforms) to one or more underlying im- 
age fragments in order to produce new images. Spe- 
cifically, the apparatus permits the application of ar- 
bitrarily defined transformations to images, including 
transformations which translate images from one 
figural model to another so thatthe "transformed" im- 
age can be further operated on in the new figural 
model. 

The user interface provided by the apparatus ex- 
plicitly represents all the underlying image fragments 
being used, all the operations being applied to them, 
and all the places where output images should be pro- 
duced. Specifically, an operator works by manipulat- 
ing a graphical flow diagram whose nodes represent 
image fragments (sources), operations (transforms), 
and outputs (sinks). This allows the reuse of image 
fragments and transformed image fragments (the out- 
puts of transform nodes) without confusion as to what 
effect changes in the sources and operations will 
have on the final output image. 

The modular construction of the apparatus 
means that it is completely extensible, both with re- 
spect to the transforms available and the underlying 
figural models from which one may draw source im- 



ages. Specifically, new transducers can be integrated 
which perform arbitrary image transformations or 
which implement new figural models. In order to inte- 
grate a new figural model, all that is required is the in- 
5 tegration of image sources from that new figural mod- 
el and operations (transforms) for converting an im- 
age in the new figural model to one of the existing 
figural models. Once an image from a new figural 
model is converted to an existing figural model, op- 
10 erations appropriate for the existing figural models 
can be performed on the converted image. 

The present invention is implemented on a con- 
ventional hardware system 100 which includes a mi- 
croprocesser 110 for receiving signals from, and out- 
15 putting signals to, various other components of sys- 
tem 100, according to one or more software programs 
run on microprocessor 110. A user input device 120 
such as, for example, a mouse, a keyboard, a touch 
screen and/or combinations thereof, is provided for 
20 permitting an operator to interface with the editor by, 
for example, supplying control signals thereto. An im- 
age output device 130, such as, for example, a video 
display monitor, or a printer, can be provided for out- 
putting image data. An image input device 140, such 
25 as, for example, a scanner, can be provided for scan- 
ning images which are then stored in memory as im- 
age sources. Image sources can also be supplied 
from, for example, discs. 

Microprocessor 110 includes a read only memory 
30 (ROM) 1 50 for storing operating programs and image 
data, used to construct and edit images. Arandom ac- 
cess memory (RAM) 160 is provided for running the 
various operating programs, and additional files 170 
can be provided for RAM overflow. 
35 One embodiment of the present invention was 

implemented in version 1.2.2 of Macintosh Allegro 
Common Lisp (Macintosh is a registered trademark of 
Apple Computers; Allegro Common Lisp is a trade- 
mark of Franz, Inc.), a dialect of Common Lisp avail- 
40 able from Apple Computer. This prototype will run on 
any computer which supports Macintosh Allegro 
Common Lisp, however, a Macintosh II computer (or 
faster) with at least five megabytes of RAM and a 
483mm diagonal monitor is preferred so as to provide 
45 an editor which is adequately fast. 

In order to convey the manner in which the appa- 
ratus of the present invention works, and the basic 
mechanisms which provide it with its novel capabili- 
ties, examples of the user interface will now be pro- 
50 vided. 

The apparatus displays an ACG which represents 
the image manipulation processes which are per- 
formed on one or more image fragments to produce 
one or more final output images. The ACG is dis- 
ss played in, for example, a window on a monitor's dis- 
play screen. When executed, ths final output images 
created by the ACG (some of ths output images may 
be components or intermediate images of other out- 
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put images) are also displayed on the monitor display 
screen in their own respective windows. Thus, the 
present invention provides a type of two-view graph- 
ics editor. One view shows the final (i.e., the WYSI- 
WYG) image(s), and the other view displays the ACG 5 
representing the construction of that (those) inv 
age(s) (i.e., the image fragments and operations per- 
formed on the image fragments which result in the 
output image(s)). 

10 

Example One 

Figure 2 shows a simple, first example of a dis- 
play screen which includes an ACG displayed in the 
lefthand window and the final output image displayed is 
in the righthand window. The final image is a triangu- 
lar portion of an image of Einstein superimposed on 
a dotted background. The ACG has three types of 
nodes; source nodes; transform (operator) nodes; 
and sink nodes. 20 

Source nodes, which are the bottom-most ("lear) 
nodes in the ACG, represent image fragments or 
other source data. In the ACG of Figure 2, the source 
nodes are labelled (from left-to-right) C,B,F,D. The 
icon for each of these nodes indicates the type of 25 
source data it contains. (Figure 9 is a list of the various 
icons which can be selected). Thus, in Figure 2, the 
node labelled C contains a uniform raster pattern (a 
"background" raster image), while node B contains a 
set of vectors, or line segments. The actual contents 30 
of each of the four source nodes are illustrated in the 
display of Figure 3. Each node's contents is displayed 
in a separate window which is labelled with the same 
letter as that node. For example, the window entitled 
"Segment Set B" corresponds to the source node "B" 3S 
in the ACG. Window B thus reveals that the content 
of node B includes three line segments intersecting to 
form a triangle. When displayed in a window on the 
monitor screen, the contents of these nodes can be 
edited by an operator inputting commands. For exan> 40 
pie, the operator can manipulate a cursor and/or 
press buttons (on a mouse or keyboard) to edit the 
contents of a source node. Additionally, source im- 
ages can be imported from external libraries in a con- 
ventional manner. The editing steps can be per- 4S 
formed in conventional ways using conventional soft- 
ware. 

Transforms (operator nodes) are the intermedi- 
ate nodes in an ACG. In Figure 2, the three icons lab- 
elled T1 , T2, and T3 are transforms. Transforms op- so 
erate on image fragments, i.e., they take image frag- 
ments and other data as input and produce a new im- 
age fragment as output. Inputs to transforms come 
either from source nodes (primitive image fragments) 
or from the outputs of other transforms (which can ss 
produce intermediate images). Thus, the transform 
T1 has two inputs, both of which are the content of 
source nodes B and F, while T2 has two inputs, one 



the content of source node D, the other the output of 
transform T1. (Source nodes can be thought of as 
transforms that have no inputs and produce their con- 
tent as output). The inputs and outputs of transforms 
are typed. Thus, T3 takes two or more raster images 
as input and produces as output a raster image con- 
structed by superimposing the input images one on 
top of the other. 

Sink nodes are the root nodes in an ACG, indicat- 
ing where an image is to be output. Sink nodes, such 
as node A in Figure 2, can represent windows or 
"viewers" on the display screen. In the present em- 
bodiment, inputs to sink nodes must be raster images. 
Sink nodes might also represent other output devices 
such as printers or files, and might be capable of ac- 
cepting types of input other than raster. For example, 
monitors could accept images in the video figural 
model, and a plotter could accept images in t he vector 
figural model. The window labelled "Viewer A" in Fig- 
ure 2 corresponds to sink node A in the Figure 2 ACG. 

Images are synthesized (or created) by travers- 
ing the image fragments and operations represented 
by the ACG with a structure walker. In the illustrative 
embodiment, image fragments flow upward (from leaf 
nodes to root nodes), are transformed and combined 
by transforms, and are displayed in, for example, win- 
dows on the display screen. 

Referring to Figure 2, the final output image, dis- 
played in viewer A, is the output of transform T3. T3 
is the superposition transform, it superimposes the 
output of transform T2 on top of the background ras- 
ter (the grid of dots) represented by source node C. 
Transform T2 is a clipping operator which clips out a 
polygonal section of a raster image. One input of 
transform T2 is the raster image to be clipped (source 
node D, whose contents is a raster image of Einstein), 
the other input is a set of vectors specifying the poly- 
gonal mask (the output of transform T1). Transform 
T1 is a simple "recognition" operator: a "polygon find- 
er". Given a set of vectors (the content of source node 
B) and a point (the content of source node F), trans- 
form T1 finds and outputs the smallest polygon en- 
closing that point. 

Images are created in three stages: 

1. ACG creation and modification. The operator 
creates an ACG graphically by inputting com- 
mands for selecting node types (from a displayed 
menu or list of node types), placing icons for 
these node types in the ACG window, and creat- 
ing links (edges) between the nodes. The user 
can add or delete nodes or links at any time. 

2. Image fragment (source node content) impor- 
tation and editing. A different type of editor is pro- 
vided for each type of source node, so that the 
contents of that source node can be edited. Each 
"source node editor" includes a window labelled 
with the same letter as that source node's icon in 
the ACG, in which the contents can be edited. 
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With reference to Figure 3, the window labelled 
"Segment Set B" permits the operator to move 
and create line segments which are used to de- 
fine a polygon. The window labelled "Point F" per- 
mits the location of a point, used by "polygon find- 5 
er" T1 to be changed. The window labelled "Ras- 
ter Constant D" permits an operator to define a 
raster image fragment by selecting from a menu 
of background patterns. Selections are made in 
a conventional manner by, for example, moving a io 
cursor and striking keys (on a mouse and/or a 
keyboard). Additionally, source node contents 
can be imported from libraries such as, for exam- 
ple, discs. 

3. Image Viewing. Once an ACG has been con- is 
structed and content created for each of its 
source nodes, the ACG can be executed and its 
results displayed. 

The ACG or source node contents can be modi- 
fied at any time by providing the appropriate control 20 
signals to the microprocessor using, for example, the 
cursor and key strokes. Figure 4 shows a modified 
ACG and the resulting images. The triangular clipping 
of Einstein now has a black border around it. and 
there is a new viewer window. Viewer E, showing 25 
three vectors forming a triangle. The Figure 4 ACG 
differs from the Figure 2 ACG In the addition of two 
new nodes -a transform labelled T4 and a viewer 
node labelled E- and three new edges connecting T3 
and T4, T4 and T1, and T4 and E, respectively. 30 

The new transform, T4, is a scan conversion op- 
erator it takes a set of vectors as input and yields a 
raster image by scan converting each of the vectors. 
Thus, the border around the clipped Einstein image 
is the result of scan converting the recognized poly- 35 
gon (which was also used as a mask to the clipping 
operator T2) and superimposing it on top of the clip- 
ped Einstein and the dotted background. The output 
of transform T4 is also displayed in Viewer E. A com- 
parison of Viewer E with the contents of the window 40 
labelled "Segment Set B" in Figure 3 shows that trans- 
form T1 has "sliced and diced" the original vectors in 
node B, outputting only those fragments that form a 
polygon (in this case, a triangle). 

Viewer A in Figure 4 now displays a window, i.e. 45 
a picture of Einstein viewed through a polygon (shap- 
ed window) displayed against a textured background. 
In Figures 5 and 6, the window metaphor is further ex- 
tended. These extensions make use of two additional 
nodes of the same type: a translation transform (T6 so 
and T7 in Figures 5 and 6) translates or displaces an 
image on the display surface. The inputs for the trans- 
lation transform are a raster image to be displaced 
and an amount (delta x, delta y) of displacement. The 
displacement nodes are source nodes G and H. 55 

In Figure 5, the superposition of Einstei ' i border 
and background has been regrouped so a tr slation 
operator can be attached to the superposition of Ein- 



stein and border. As a result, the displacement node 
G acts as a "window mover". By changing the x and 
y values in displacement G (in the editor window for 
the content of node G) and re-executing the ACG, 
Einstein plus border can be displaced (moved) by del- 
ta x and delta y. 

In Figure 6, the same basic move, the addition of 
translation and displacement nodes in the appropri- 
ate places creates yet another effect, i.e., displace- 
ment node H acts as a scroll bar. Entering x and y val- 
ues in displacement H (attached to transform T7) and 
re-executing the ACG causes just the Einstein raster 
image to be displaced (scrolled) by delta x and delta 

y- 

Figure 7 illustrates how the shape of the window 
can be changed by simply editing the vectors from 
which the window was originally formed and rerun- 
ning the ACG. The structure of the ACG in Figure 7 
covered by the Segment Set B Window is the same 
as the Figure 6 ACG. 

Example Two 

Figure 8 illustrates what an editor screen might 
look like when an operator has finished constructing 
a more complex appearance. As in the example of 
Figures 2-7, the center piece of the user interface 
provided by the present invention is the large graph 
view window 10 which is where an operator pieces to- 
gether the desired appearance by combining image 
fragments and transforms in a desired way. 

The icons used to depict nodes in the ACG are 
based on the type of the node, rather than its con- 
tents. For instance, two nodes containing raster im- 
age fragments will be depicted with identical icons 
even though the image fragments they contain might 
be different (although they will be labeled with a dif- 
ferent letter). Similarly, the nodes for two different ap- 
plications of a masking operation will be depicted with 
identical icons because they are both masks. It is im- 
portant, when viewing an ACG not to confuse the ap- 
pearance of the icon for a node (which indicates its 
type) with the appearance output by the node. 

Figure 9 shows some different icons which can 
be used and the types of nodes they represent. Icon 
20a Is a viewer node. Icon 20b is a background node. 
Icon 20c is a raster appearance. Icon 20d is a line seg- 
ment appearance. Icon 20e is a displacement icon. 
Icon 20f is a point icon. Icon 20g is a superposition 
icon. Icon 20h is a translation icon. Icon 20i is a scan 
conversion and parameterized scan conversion icon. 
Icon 20j Is a polygon finder icon. Icon 20k is a mask 
icon. 

Note that square nodes which denote input 
source images and output constructed images, are 
labeled with single letters from "A" to "Z". These lab- 
els are used for identification. Whenever an operator 
requests, by inputting a command, such as with the 
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operation of a cursor and the stroking of keys, via a 
mouse and/or a keyboard, the graphics editor of the 
present invention will display a special window which 
shows the contents of any square node. This win- 
dow's title will contain that node's letter. In the case 5 
of input source images, these windows are editors 
which can be used to alter the node's image data con- 
tent In the case of output images, these windows are 
viewers only, as the image is completely determined 
by the ACG itself. For this reason, output nodes are io 
referred to as "viewer" nodes. Figure 8 contains a 
number of viewer and editor windows to the left of the 
graph view window 10. 

The present invention can be used in a conven- 
tional way, i.e., by mousing in windows, and by invok- is 
ing commands from menus. Additionally, a prompt 
window 12 can be provided to give feedback on the 
course of operations invoked. 

In order to provide a better understanding of the 
manner of which the ACG 10 of Figure 8 produces the 20 
contents of the viewers in Figure 8, more simplified 
ACG's will first be described. Figures 1 0A and 1 0B are 
the simplest possible ACG's, since each simply con- 
nects an image fragment to a viewer. The ACG of Fig- 
ure IDA is a viewer connected directly to a "back- 25 
ground", i.e., an image fragment containing a raster 
pattern which is repeated horizontally and vertically 
to fill its allotted space in the final appearance. The 
ACG of Figure 1 0B is a viewer connected directly to 
a raster appearance, i.e., an Image fragment contain- 30 
ing a previously-prepared bitmap. Raster appearanc- 
es can comprise previously scanned images which 
are stored in a file (library). Figures lOAand 10B il- 
lustrate that the present invention can be utilized to 
output the contents of image sources directly, without 35 
any transformation operations being performed 
thereon. 

To construct ACG's in the graph view window, 
commands are issued (e.g., with mouse clicks) to in- 
dicate where nodes are desired. Similarly, a com- 40 
mand can be issued to indicate that two nodes should 
be connected. This can be accomplished, for exam- 
ple, by dragging a cursor between two nodes while 
pressing a button on a mouse and a key on the key- 
board to form an edge between the two nodes. When 4S 
two nodes are connected by an edge, the one on top 
is considered the parent and the one below the child. 
Parent nodes work by taking the outputs of their chil- 
dren as inputs and then producing outputs thenv 
selves. Thus, each of the ACG's in Figs. lOAand ICS 50 
would require simply two clicks and a drag to con- 
struct. The viewers would take the outputs of the im- 
age fragments as inputs and display them on sepa- 
rate windows. 

Thus, the viewed nodes can have their contents 55 
displayed directly on the screen (in viewer windows). 
Since the sween uses a raster imaging model, the 
viewer nodes must always have a raster appearance 



(a bitmap) as their input. 

This restriction leads to Figure 11, which shows 
a viewer connected so as to display a "line segment 
appearance", i.e., an image fragment containing 
some Euclidean line segments. Since viewers can 
only display raster appearances, the connection be- 
tween the viewer and line segment appearance must 
be mediated by a "scan conversion" (the circular node 
20i) which produces a raster appearance correspond- 
ing to the line segment appearance. 

The ACG of Figure 11 is particularly interesting 
since it contains a transform node. Transforms are 
functional image-transforms which are used in the 
graph whenever it is desired to alter or combine vari- 
ous image fragments (or transformed image frag- 
ments). In the example of Figure 11, the operator de- 
sired to view line segments. However, the line seg- 
ments are drawn using a Euclidean f igural model, and 
in the present example, the viewers can display only 
appearances drawn using a raster figural model. Ac- 
cordingly, transform 20i, which transforms Euclidean 
images into raster images (a scan converter) is 
placed between the viewer node and the source im- 
age node to produce an ACG which views the desired 
appearance. 

Transforms can be used to perform many func- 
tions. They are used to transform images within a sin- 
gle figural model, to convert images from one figural 
model to another, to extract features from images for 
use in other images, and to combine images from the 
same or different figural models. Transforms serve 
both as the tools used to cut images apart and the 
glue used to put them back together. The scan con- 
verter 20i of Figure 11 is conceptually (and computa- 
tionally) a simple transform. However, the present in- 
vention permits arbitrarily complicated transforms 
which embed arbitrarily complex computations to be 
incorporated. The only requirement on transforms is 
that they be functional in the programmatic sense, 
i.e., their output must be determined solely by their in- 
puts without respect to any internal state. The exten- 
sibility of the present invention comes from the ability 
to add new transforms. 

To illustrate further the function of some simple 
transforms, reference is made to Figures 12A and 
128. Figure 12A illustrates a line segment appear- 
ance which is scan converted and then viewed. How- 
ever, this time the scan converter is parameterized, 
i.e., it takes additional inputs which determine the 
size of the scan converted lines and the background 
pattern that they are drawn with. Thus, the scan con- 
version node has three children: the line segments to 
be converted, the background pattern to use when 
drawing the rasterized lines, and a point (i.e.. an x,y 
pair) which determines the size of the rasterized 
lines. 

The ACG of Figure 128 illustrates a network 
where one transform's output is the input of another. 
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In this network, a polygon is extracted (registered) 
from an arrangement of line segments and then is 
used to mask a background pattern. The ACG of Fig. 
1 2B Is similar to a portion of the Fig. 8 ACG in that a 
polygon is found using image sources A and B, the 5 
found polygon is scan-converted, and then input to 
viewer node C. The transform 91 is the "polygon find- 
er". It takes a line segment input (from the source lab- 
eled A) and a point input (from the point source B), 
searches outwards from the point in the line seg- io 
ments to find an enclosing polygon, and then outputs 
that polygon (in this case, the polygon found is a 
quadrilateral). Transform 95 is a "mask" (or "clipper"). 
It takes polygon input (from transform 91) and a raster 
appearance image (from the source labeled F) and is 
produces as its raster appearance output the result of 
clipping the raster appearance to the interior of the 
polygon. The output is then displayed by viewer node 
C. 

Any number of different types of image sources 20 
can be provided. Some examples include raster ap- 
pearances, line segment appearances, background 
patterns, points (whose x- and y-components are al- 
ways positive), and displacements (points whose 
components can be negative). Once an image frag- 25 
ment is created - by selecting an image and (option- 
ally) editing, the image fragments can be combined 
and altered any number of ways, depending on the 
types of transducers available. 

Now, with reference to Figure 13 (which is an en- 30 
larged view of the Figure 8 ACG), the ACG illustrated 
therein and its constructed appearance will be descri- 
bed. The graph will be considered section by section 
to see how it builds the final appearance. At the lower- 
left of the ACG is a polygon-producing duster, similar 35 
to the one shown in Figure 1 2B, consisting of line seg- 
ment appearance A, point B and a polygon finder 
transform T10. The output of this cluster Is used in a 
variety of places, so to make it clear what the found 
polygon looks like, a simple scan-conversion trans- 40 
form Til Is inserted to render the polygon onto viewer 
C (see Fig. 8). The editor windows 20 and 22 are illu- 
strated to the left of the graph view window 10 in Fig- 
ure 8. Editorwindows for segments A(window 20) and 
point B (window 22) as well as viewer C are illustrated. 45 
Above and to the right of the polygon producing clus- 
ter (see Fig. 13), its output Is used both by a parame- 
terized scan-conversion T1 2 and by a mask T1 3. sim- 
ilar to the one shown in the examples of Figures 1 2A. 
and 12B. The outputs of these two transforms are 50 
then superimposed one on the other (by a superpo- 
sition transform T14) to produce a combined output 
which is both shown in Viewer F and used again as a 
piece of the final, constructed appearance in Viewer 
G. 55 

The outputs of the parameterized scan- 
conversion T12 and the mask T13 are also used sep- 
arately as the inputs to translation transforms T15, 



T16, T1 7 which each use a displacement input to de- 
termine how far to translate their source appearance. 
Finally, the output of these translations are then re- 
combined with an underlying background and the su- 
perposition discussed via superposition transform 
T18 above to produce the complete appearance of 
viewer G. 

As can be seen from the above-described exam- 
ple, the present invention, through the use of an 
ACG-based interface, provides numerous advantag- 
es over previous graphics editing systems. It is easy 
to use multiple image sources without their interfering 
with one another Image fragments can be drawn 
from a variety of f igural models. Transformations be- 
tween models are explicit and consistent. Image com- 
ponents can be reused several times in the same im- 
age, both directly and after transformation, without 
copying or confusion as to which is the original. Trans- 
formed images are no different In status from original 
image fragments - both can be used in exactly the 
same way. Any collection of component images used 
in the construction of another image can be viewed 
separately in order to further clarify their contribution 
to the image as a whole. 

There is nothing particularly interesting about 
ACGs by themselves - they are simply expression 
graphs such as those found in the abstract syntax 
trees provided by standard parsers. However, the use 
of ACGs as a tool (user Interface) for constructing and 
editing images overcomes the problems, discussed 
above, which persist in existing graphics editors. 
ACGs are directed, acyclic graphs (DAGs) whose 
nodes are called transducers. There are three basic 
types of transducers: sources, transforms, and sinks. 
Sources are image generators: they take no children. 
Transforms are image functions: they take the out- 
puts of their children as inputs and produce a single 
output which results from combining and transforming 
these inputs. Sinks are image consumers: they can 
appear only as roots of the ACG. 

ACG's typically consist of one or more source 
nodes, which could appear as children of one or more 
transform nodes, which could be children of one or 
more sink nodes. The sink node in this example, i.e.. 
the viewer, consumes an image formed in the ACG 
and automatically displays the image in a window 
whenever the ACG is evaluated. Thus, viewer nodes 
are used to specify exactly which of the source or 
transform nodes in the graph are desired to be 
viewed. Since the ACGs are DAGs, not trees, an op- 
erator can both view the output of a node and use that 
node as the input of other (transform) nodes. 

The typical construction process for an ACG In- 
terleaves the construction and connection of nodes 
with the editing of source nodes and the evaluation 
of the graph to produce output images. Feedback 
from the evaluation process is then used as a basis 
for structural alterations in the graph and further edit- 
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ing of the source images. Thus, the process of ACG 
construction and revision is analogous to that of pro- 
gram construction and revision, which is natural since 
ACG's can be thought of as programs (or expres- 
sions) which produce images. 

The edges in an ACG run from named, typed out- 
put ports on one node to named, typed input ports on 
another. Only one edge can run between any two giv- 
en ports. The type of an ACG node determines the 
number, names and types of its ports. The type of an 
input port determines the allowable types of output 
ports it can be connected to. Additionally, input port 
type determines whether the port is unary (can ac- 
cept only one incoming edge) or n-ary (can accept 
any number of incoming edges ordered in some se- 
quence). Output ports can always be connected to 
any number of outgoing edges; their type is used only 
to determine which input ports those edges can be 
connected to. A well-formed ACG is a DAG all of 
whose ports have at least one edge connected to 
them and all of whose edges obey the type restric- 
tions of their ports. 

In the present embodiment, the ACG framework 
is a single mechanism which provides both a user In- 
terface that makes complex appearance construction 
precise and convenient and an implementation model 
that makes figural model extensibility and integration 
precise and convenient. 

The user interface presents a simple "erector set" 
view of images. Images are readily seen to be con- 
structed from small, reusable image fragments and 
operators. This scheme has three important features. 
First, it allows for the precise specification, modifica- 
tion, and viewing of the structure of complex images. 
Second, it allows underlying commonalities across 
domains (figural models) to be easily captured and re- 
used. Third, it provides a simple and convenient inter- 
face to extensibility. 

Current WYSIWYG graphics editors have devel- 
oped a number of features aimed at allowing the user 
to navigate through and manipulate components of 
complex images. Grouping and ungrouping of image 
elements, locking figures in place on the surface, the 
use of multiple layers (or planes) on which figures can 
be distributed, the ability to hide figures (make them 
temporarily invisible) or to magnify them. Mecha- 
nisms such as these have been developed, in part, 
because the WYSIWYG discipline makes it hard to 
see, and therefore to manipulate, the structure of 
complex images. By providing a view in which the 
structure of an image is readily apparent, the present 
invention makes it easy for the user to see and refer 
to the elements. of the image. 

Even though previous techniques such as group- 
ing, locking and layering are of some use in manipu- 
lating images with many elements, they provide no 
help to the user in getting at the relations among ele- 
ments. For example, none of these techniques pro- 



vides any help if an image has shared elements (as 
in Figure 4, where the polygon serves both as a clip- 
ping region and as a border). For another example, 
consider creating a 'bitmap polygon* by scan- 
5 converting a vector polygon (as can be done in some 
commercial graphics editors, such as SuperPaint®). 
Once the scan-conversion is through, the relation- 
ship between the figures is lost: changes made to the 
vector polygon have no effect on the bitmap. Contrast 
10 this behavior with that of Figure 7. 

The display of ACGs is not biased toward partic- 
ular figural models: it works equally well for raster 
graphics as for vector graphics ~ and for text video, 
or 3D graphics. Nor is it biased toward the details of 
IS particular models — to the presence or absence of 
particular operators. It therefore provides a conve- 
nient interface to the editor's extensible graphics en- 
gine. Thus, the same basic mechanism (ACGs) that 
allows for extensibility suggests a simple and power- 
20 ful interface to that extensibility. 

For purposes of illustration, windows and menus 
for an exemplary editor will now be provided. 

An operator provides commands, for example, by 
using screen displayed menus and a mouse. The 
25 graph view window is displayed at all times. Addition- 
ally, a prompt window can be displayed to provide the 
operator with feedback. The first menu, the Ops 
menu - which is always enabled - contains commands 
which apply to the system operation as a whole. The 
30 Graph Ops menu - which is enabled only when the 
graph view window is active - contains commands 
which apply to the graph view window. The exact con- 
tent of the menus and the use of the graph view win- 
dow are briefly described below. 

35 

The Prompt Window 

The prompt window is used by all system com- 
mands whenever they wish to prompt t he operator, In- 
40 form an operator of their progress, or warn an oper- 
ator of an error. The prompt window cannot be select- 
ed since it is intended for output only. 

Viewer and Editor Windows 

45 

Two other types of windows can be created and 
destroyed by user request. Viewerwindows show the 
images input to viewer nodes. The contents of these 
windows cannot be edited directly, rather an operator 

50 must edit and re-execute the ACG to cause the viewer 
windows to be filled with the desired contents. Viewer 
windows are created automatically whenever the 
ACG is executed. One viewer window is created for 
each viewer node, and preferably, the title of the win- 

55 dow (which matches the title of the node) is associ- 
ated with that window. These windows can be dosed, 
however, they will reappear the next time the ACG is 
evaluated. To cause a viewer window to be deleted 
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permanently, the viewer node is merely deleted from 
theACG. 

Editor windows are created when an operator is- 
sues a command to edit a source node. Each source 
node has its own type of editor. Editor windows are ti- 5 
tied to match their sources (as with viewers). Editor 
windows are closed when the editing functions are 
completed. Some editor window types can have spe- 
cial menus which get added to the menu bar when an 
editor window of that type is created. These menus ro 
will be enabled only when an editor window of that 
type is active, and will go away if all the editor win- 
dows of that type are closed. The specific editors can 
be implemented using techniques in the prior art. 
Since all the sources can be imported, commercial, is 
existing editors can be used. All that is required is to 
provide a means for translating the user input conr>- 
mands to commands which can be understood by the 
software. This can be accomplished in a conventional 



manner. 



The Graph View Window 



Figure 9 is a list of the icons used in the graph 
view window for the various types of nodes. The in- 
put/output protocols for these nodes will now be de- 
scribed. 

Sink Nodes 

As mentioned above, one type of sink node is the 
type viewer 20a. Viewer nodes can have only one 
child, and their function is to display the output of that 



20 



The Ops Menu 

The Ops menu permits an operator to select, for 
example, the following (and other) commands which 25 
effect the state of the editor as a whole: 

"Edit Graph" makes the graph view window the 
active window. The editing operations available in 
this window are described below. 

"Execute Graph" evaluates the ACG in the 30 
graph view window, and updates all viewer windows 
appropriately. Progress of the command is shown, for 
example, in the Prompt Window, as are reports of any 
errors encountered while executing the graph. 

"Save ACG" prompts for a file in a directory (or 35 
in a disc file) and saves the current ACG to that file 
for later reuse. 



40 



The graph view window is used to construct and 
edit the ACG. In this window, commands can be is- 
sued to create, remove, connect, and disconnect 
transducer nodes. These commands can be issued, 
for example, using mouse clicks and drags. 45 

Node Types 



50 



55 



10 



child, which, in the present examples, must be a ras- 
ter image. Viewer nodes have no outputs, and so can- 
not be the children of other nodes. Whenever an ACG 
is executed, a viewer window is created for each view- 
er node in the ACG. This viewer window displays the 
input of the viewer node. Both the node icon and the 
viewer window are labeled with, for example, a letter 
from A to Z so it is clear which viewer node's inputs is 
being displayed in each view window. Other types of 
sink nodes could, for example, send images to files 
for storage, or to a printer (or plotter) for printing. 

Transforms 

Each transform type is characterized by its input 
(their type and cardinality) and how its output is gen- 
erated from its input Af irst type of transform is the su- 
perposition transform 20g. This transform takes any 
number of children. The output of each child must be 
a raster image. The outputof the superposition (a ras- 
ter image) is obtained by superposing (layering) the 
outputs of the children in order, so that the inputs of 
later children obscure (are layered on top of) the in- 
puts from earlier children. The order of the children is 
determined by the order in which the superposition is 
connected to its inputs, however, other ordering tech- 
niques can be used. In this example, the child which 
is connected first gets layered first and is covered by 
the child connected second, which is covered by the 
child connected third, and so on. 

A second type of transform is the scan conver- 
sion transform 20i. This transform takes a single 
child, which must be a line-segment vector image, 
and produces as its output a raster image obtained by 
scan-converting the line segments with a width of 
one pixel and a black pen pattern. 

Another type of scan converter is a parameter- 
ized scan converter. This transform takes three chil- 
dren. One must be a line-segment vector image, an- 
other must be a point source, and the third must be 
a background source. The order of the children does 
not matter. The output of the transform is a raster im- 
age obtained by scan converting the line segments 
with a pen whose width is the x-value of the point 
source, whose height is the y-value of the point 
source, and whose pattern is the background pattern. 
The default values for the x and y components can be, 
for example, 1, while the default background pattern 
can be black. 

These two types of scan converters could be of- 
fered as separate choices, or as a single choice (with 
the default values for background and pen-size oper- 
ating as the non-parameterized scan converter). 

A third type of transform is the polygon-finder 20j. 
This transform takes two children, a line segment im- 
age and a point source, in any order. Its output is a line 
segment image whose line segments form a polygon. 
The output polygon is the smallest polygon which can 
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be registered in the line segment image that contains 
the point whose coordinates are given in the point 
source. If there is no such polygon, then evaluating 
the polygon- finder node will produce an error. 

A fourth type of transform is a mask 20k. This 5 
transform takes two children, a raster image and a line 
segment image (whose segments should form a poly- 
gon), in any order. The output of the mask is a raster 
image consisting of the input raster clipped to the in- 
terior of the polygon. to 

Af ifth type of transform is a translation transform 
20h. This transform takes two children, a displace- 
ment source and an image (either a raster image or 
a line-segment image), in either order. Its output is the 
input image translated by the displacement. »5 

Source Nodes r 

Source nodes, as mentioned above, produce an 
output but take no inputs. Thus, source nodes have no 20 
children. Each source node, when first created, for ex- 
ample, can be given a default output value (contents) 
based on its type. The source node can then be edited 
in order to change its contents. Similarly, source im- 
ages can be imported firom external files (e.g. discs) 25 
to produce the content of a source node. 

The display icon for each source node in an ACG 
is labeled with, for example, a letter from "A" to "Z°, 
so that source nodes can be distinguished easily. If 
an operator asks to edit a source node, an editor win- 30 
dow is created and labeled with that node's letter, so 
it is always clear which editor window goes with which 
node. 

Different source types use different types of edi- 
tor windows. The editor window for each source type 35 
is described with the source type. 

A first source type is a point 20f. A point is an or- 
dered pair (x,y) of non-negative, integer coordinates. 
The default value for a newly created point node can 
be (0,0). Points are edited using a dialogue window 40 
t hat has a text item for each of the coordinates, an OK 
button, and a cancel button. The window labeled Point 
F in Figure 3 is a point source editor window. The x 
and y coordinate values can be edited by entering 
new values and buttoning OK. 45 

A second type of source Is a displacement node 
20e. A displacement is an ordered pair (x,y) of inte- 
gers (either negative or positive). The default value 
for a newly created displacement node can be, for ex- 
ample, (0,0). Displacements are edited exactly as 50 
points are. 

A third type of source is a background source 
20b. A background is a raster image of indefinite size 
filled with, for example, an 8 pixel square repeating 
image called a pattern. Backgrounds are taken to ex- 55 
tend infinitely in all directions, i.e., they will complete- 
ly fill any size pen (when used for scan conversion) or 
mask. The default pattern for a newly-created back- 



ground node can be, for example, all white. Back- 
grounds are edited in a special dialogue window 
which permits an operator to choose from a variety of 
predefined patterns. The window labelled Back- 
ground C in Figure 3 illustrates a window displaying 
some illustrative patterns. The selected background 
pattern is indicated by enclosing it in a black square. 
The background pattern can be changed by buttoning 
a different pattern and then buttoning OK. Buttoning 
cancel will move the black square back to the original 
background pattern. 

A fourth type of source node is a raster appear- 
ance 20c. A raster appearance is a previously pre- 
pared raster image, i.e., a bitmap. Raster appearanc- 
es are edited using a special dialogue window which 
allows an operator to choose from a variety of pre- 
pared raster images. The window labeled Raster Con- 
stant 0 in Figure 3 is an example of the special dia- 
logue window for raster appearances. The selected 
raster image is highlighted by, for example, placing a 
box around it. The images are described rather than 
shown, although a display which shows the images 
could also be used. An operator can change the raster 
appearance's bitmap by buttoning a different" name 
and buttoning OK. 

A fifth type of source is a line segment appear- 
ance 20d. A line segment appearance is a set of Eu- 
clidean line segments, i.e., a set of infinitely thin line 
segments whose end points have real coordinates. 
Editing a line segment appearance will now be descri- 
bed. 

As described earlier, line segment appearances 
cannot be viewed directly, rather they must always be 
scan converted and the results viewed. Accordingly, 
the operating system has an editor window for a line 
segment appearance node display of the result of ap- 
plying a scan-conversion transform to the node seg- 
ments. The window labeled Viewer C in Figure 8 illus- 
trates such a conversion. The editing operations in 
line segment appearance editor windows are invoked 
from a menu that is enabled whenever such an editor 
is active. When an operator creates a line segment 
appearance node in the graph view window, that 
node is automatically connected to a scan conversion 
node and a viewer node, and the editing window for 
that viewer node is then displayed so that the oper- 
ator can edit the line segment appearance. The line 
segment appearance can be edited any number of 
ways which are conventional in the art. New seg- 
ments can be created, existing segments can be de- 
leted, and any segments can be moved. 

Implementation 

The present invention is implemented in a man- 
ner which allows for flexible, precise, simple integra- 
tion of new figural operations and figural models 
(graphical domains) into existing operations and 



Copied from 10368583 on 01/09/2006 



21 



EP 0 528 631 A2 



22 



models. Programmers can add new operations for 
use with existing models (e.g., affine transformation 
over vector art), new operations between existing 
figural models (e.g., raster-to-vector recognition op- 
erators, vector-to-raster scan-conversion opera- 5 
tions), and entirely new figural models (e.g., text, 3D, 
video) both with their own operations and with exist- 
ing-model inter-conversion operations. 

The extensibility of implementation, just like the 
precision of the user interface, is rooted directly in io 
ACGs. The key to this is the observation that ACGs 
are basically data flow diagrams. The basis of the im- 
plementation of the present graphics editor is the di- 
rect representation of ACGs in an object-oriented 
graph representation language. For example. Com- is 
mon Lisp Object System (CLOS) graph representa- 
tion language can be used. Nodes in the graph are 
objects which keep track of their connections to 
neighbors and which respond to a standard set of 
graph traversal messages. The graphic functionality 20 
of transducers is implemented by making their objects 
inherit functionality not only from the node dass (i.e., 
the type of input/output-source, transforms, sink) but 
also from a graphics operator class (i.e., perform spe- 
cific function based on input) which responds to stan- 25 
dard operation messages such as reading inputs and 
providing outputs. Transducer node types (all node 
types are illustrated in Fig. 9) correspond to subclass- 
es of the generic operator class which implement 
functionality appropriate to the node type. 30 

As a user uses the graph-construction interface 
to build an ACG, the editor is in fact building this ACG 
in the underlying object model. When the operator 
asks that the ACG be executed, a graph traversal al- 
gorithm is run which, as it walks the graph from the 35 
bottom up (from leaf nodes to root nodes), invokes the 
graphics operations associated with each source, op- 
erator, and sink. This has the effect of constructing 
and displaying the appropriate appearance. In other 
words, the present invention executes ACGs by prop- 4o 
agating graphics information along the data flow arcs 
which connect the ACG nodes. 

This very simple, direct implementation model 
has two great advantages. First, as mentioned early, 
it allows for the easy integration both of new opera- 4S 
tors and of new figural models. Second, it allows tra- 
ditional compiler optimization technology, already 
adapted for use with abstract syntax graphs, to be 
employed directly so aS to reduce both the time and 
space requirements for graph execution. 50 

If a programmer wished to integrate some new 
figural functionality, expressed perhaps as a library 
of routines and some associated data structures, two 
aspects must be addressed. Appropriate access must 
be provided to the operation library, and appropriate 55 
typing information must be provided to the ACG con- 
struction interface. However, since in the present in- 
vention, both operation invocation and graph con- 



struction are handled via the same mechanism, one 
achieves both simply by writing some object-oriented 
"glue" interfaces to the library. (And, possibly if one 
is greatly concerned with efficiency, some optimiza- 
tion advice to the graph walker) 

The implementation of the present invention is 
best understood as being broken into four intercon- 
nected modules represented by a block diagram in 
Fig. 14: 

1) ACG support module 1410. This module pro- 
vides a representation of the ACG graphs, speci- 
fies the input/output protocols for the transducers 
embedded in ACGs, and implements execution of 
ACGs. 

2) Graphics support module 1420. This module 
implements the figural models which are sup- 
ported by the graphics editor. 

3) Transducer module 1430. This module pro- 
vides the implementations of the particular trans- 
ducers used in the graphics editor. 

4) User interface module 1440. This module pro- 
vides for the editing of ACGs and image frag- 
ments, and for the display of generated images. 

ACG support 

The ACG module supports two related views of 
ACGs: one - crucial to the user interface, as directed 
acyclic graphs whose nodes are transducers; another 
- crucial to image generation - as expressions in a 
functional imaging language whose operators are 
transducers. Thus, an ACG can be manipulated as a 
directed, acyclic graph (DAG) - creating and destroy- 
ing transducer nodes, adding and removing edges be- 
tween nodes - and then asking the editor to produce 
one or more appearances from that graph by evalu- 
ating the expression it represents - running each 
transducer in the graph in the proper sequence over 
the proper inputs. 

The ACG support module represents ACGs di- 
rectly as DAGs. The expression view is supported by 
a graph traversal (evaluation) algorithm which recur- 
sively descends an ACG from the roots to the fringes, 
each node in the ACG is evaluated by applying its 
transformation to the inputs obtained by evaluating 
its children. As an operator manipulates the graph 
view in the user interface, that module constructs an 
appropriate ACG using the tools provided by this 
module. Then, when an operator asks to see the im- 
age produced by the ACG, the user-interface module 
uses the evaluation traversal to produce an output inrv 
age at each of the roots of the prepared graph, and 
then displays those images. 

The representation of ACG DAGs makes use of 
a special metaclass (i.e., "graphic-operator "). In- 
stances with metaclass "graphic-operator* act as 
nodes of ACGs. The nodes can have an arbitrary 
number of named ports each of which can contain 
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children, and they keep back pointers to all their par- 
ents. All the standard graph operations are defined 
over nodes by defining methods over the special 
class "operator" which is autonnatically made a super 
class (i.e., raster, vector, etc.) of any instance with s 
metaclass "graphic-operator". These operations In- 
clude both queries (How many children does this 
node have? What are its port names? etc.) and ma- 
nipulations (add this child to this port of this node, 
etc.). 10 

The evaluation traversal for ACGs works by in- 
voking an evaluation method (i.e.. "perform-draw-ac- 
tions") on the roots of the ACG. This method is spe- 
cialized on each class of transducer so as to perform 
the operations appropriate to that class. Any trans- is 
ducer which takes inputs (has children in the graph) 
will invoke "perforn^v draw-actions" recursively on its 
inputs. This evaluation proceeds post-order from the 
roots to the fringes in an ACG. Thus, a parent will not 
be evaluated until all of its children are evaluated. 20 

This evaluation algorithm invokes "perform- 
draw- actions" on a node at least once for each parent 
that the node has. Since ACGs are DAGs, rat her than 
trees, this may lead to nodes being evaluated more 
than once during a single evaluation pass. Since 25 
transducers are required not to have side effects, 
however, this repetitive evaluation cannot affect the 
output produced by the evaluation process, only the 
efficiency of evaluation. Since efficiency must be of 
concern in any real-time editor, ACG evaluation can 30 
be optimized. 

One class of optimizations is suggested by the 
observation that while the present Invention eval- 
uates the graphic-language expression represented 
by an ACG, it would be possible instead to compile 35 
such an expression into a more efficient form, and 
then execute that. If the additional cost of compilation 
were sufficiently small, then the combined cost of the 
compilation and successive execution would be 
smaller than that of the original evaluation. Three 40 
natural optimizations suggested by this approach 
are: 

1) Repeated-expression removal. As observed 
above, repeated evaluations of the same sub- 
graph of an ACG must produce identical results. 45 
In fact, repeated evaluations of isomorphic sub- 
graphs must also produce the same results 
(where two fringe nodes are considered equiva- 
lent if they contain the same image fragments). 
After analyzing the expression graph and picking 50 
out isomorphic subgraphs, evaluation of the sut>- 
graphs could be done just once and their output 
cached for reuse as necessiary. 

2) Unreachable-expression removal. Not all the 
output of a node (or subgraph) is always used by 55 
the parent of that node. Pre-analysis of the ACG 

can reveal such situations, allowing the removal 
of unused sub- expressions and a consequent 



savings of execution time. 
3) Idempotent-expression removal. Sometimes 
the output of a transducer is identical to its input. 
Pre- analysis of the ACG can reveal such situa- 
tions, allowing the removal of such transducers 
and a subsequent saving of execution time. 
Implementation of these three optimizations can 
be achieved by augmenting ACGs so that they are at- 
tribute- graphs (such as those discussed In the stan- 
dard compiler- theory literature) and then doing spe- 
cialized attribute propagation as a precursor to the 
evaluation traversal. 

One more optimization-incremental reevalua- 
tion- could be added to the ACG module in order to 
improve the response to changes made in an existing 
ACG. Since an ACG is often reevaluated by a user af- 
ter making changes in small pieces of that ACG, and 
since unchanged subgraphs will always produce the 
same output on subsequent evaluations, reevalua- 
tions could often be speeded up by not re-evaluating 
unchanged portions of the ACG. 

Graphics support 

In the present embodiment, in order to support 
the Euclidean models, the software must provide ru- 
dimentary support for the finding of segment-point 
distances and segment-segment intersections. The 
raster model, and conversions from the Euclidean to 
the raster models can be supported, for example, by 
QuickDraw®, a proprietary set of routines made 
available for general use on the Macintosh by Apple 
Computer. 

Transducers ' 

Transducers have two types of functionality. They 
are the nodes in ACGs, thus they have a graph func- 
tionality which involves keeping track of their children 
and their parents. They are also appearance transfor- 
mers, thus they have a graphics functionality which 
involves manipulating the appearances of their chil- 
dren so as to produce an output appearance. 

As explained above, the graph functionality of 
transform instances is implemented by making them 
have a metaclass operator-class. This gives each 
transform instance one slot for each of its named 
ports: these slots keep track of the instance's chil- 
dren, and graph operations on the instance work by 
making changes in the slots. 

Also as explained above, the graphic functional- 
ity of transform instances is implemented by defining 
special evaluation methods over them. That is, for 
each class of node, a method definition for'perform- 
draw-actions" is defined which performs a specified 
function on a specific class (or classes) of input to cre- 
ate a specific type of output 

A distinction is made between transducers that 
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implement fringe nodes of ACGs (which are image 
fragments) and those that implement internal nodes 
(transforms). Since the graphic functionality of trans- 
ducers is determined by method definitions, and 
since method definitions are class-based, the class 5 
of a transducer is the primary determiner of its behav- 
ior. What differentiates the behavior of instances of 
the same transducer class is the local state of that in- 
stance, namely its slot contents. We have already 
seen that each transducer stores its children in its to 
slots, thus, the behavior of internal nodes (trans- 
forms) can be made dependent on their children. In 
fact, the implementation of internal transducers 
(transforms) are consistently dependent only on their 
class and their children. Thus, they are functionally is 
defined. 

Fringe nodes (image sources), on the other hand, 
have no children, and so instances of the same class 
cannot be differentiated thereby. Instead, each fringe 
transducer is given an internal state which can be 20 
thought of as an image fragment in one of the figural 
models, and the graphic functionality of the transduc- 
er is determined by that image fragment. This is why 
the user interface provides editors for fringe nodes, 
but not for internal nodes. Internal nodes have no 25 
state to edit other than their children, while fringe 
nodes do. 

The important thing about the transducer frame 
work is that it makes building extensions very modu- 
lar. The graphic functionality desired can simply be 30 
couched as methods attached to a particular class. 
The input and output of this functionality are then au- 
tomatically attended to by the ACG module. 

User Interface 35 

The user interface breaks into three parts: ACG 
editing, image-fragment (or source) editing, and view- 
ers. Each part is extremely straightforward. 

The ACG editing mechanism works by interpret- 40 
ing signals input by a user (for example, by using 
mouse clicks and drags) as step-by-step instruction 
for building a DAG. (Other means for interpreting op- 
erator input commands, of course, can be provided.) 
It then follows these instructions two-fold: it builds a 45 
special interface DAG whose nodes and edges keep 
information about the graph window's display, and it 
also uses the ACG module to build the appropriate 
ACG. This strategy of keeping a separate interface 
DAG (rather than annotating the DAG with display in- so 
formation) was adopted to increase the implementa- 
tion's modularity. Major changes can be made in both 
the user interface DAG representation and the ACG 
DAG representation without any interference be- 
tween the two. 55 

The source editing mechanism consists of unre- 
lated, dedicated editors, one for each type of source 
(line segments, prepared bit maps, backgrounds. 



points, and displacements). Four of the five editors 
can be, for example, Macintosh-style dialogues which 
allow a user to choose from a variety of pre-known 
possibilities. 

The line-segment editor, however, recursively 
uses the transducer mechanism of the present inven- 
tion. The basic problem in building an editor for line 
segment appearances which are defined using a Eu- 
clidean model, is that the editing interface -the dis- 
play screen - uses a raster model. This problem must 
be faced by all editors whose input and output device 
models differ from the figural models of the appear- 
ances they edit. Thus, there is no way to display the 
appearance being edited directly , nor is there any 
way to use mouse pointing to specify directly end 
points for segments. Both the input and output of seg- 
ments must make use of some raster representation. 

The line-segment editor of the present invention 
handles this problem by recursively using the trans- 
ducer network. When a user wishes to edit a line seg- 
ment image source, a small ACG consisting of that 
source node, a single scan-conversion transducer, 
and a viewer as described above, is created. The 
viewer of this ACG is used as the viewer of the seg- 
ment editor, thus the user sees the same represen- 
tation of the segments as they would if they used a 
scan-converter transducer in the ACG. For purposes 
of referring to segments, the coordinate system of the 
line segment domain and the coordinate system of 
the raster domain are kept synchronized, so that 
mouse clicks appearing in the raster domain can be 
interpreted directly as mouse clicks in the vector do- 
main. 

Figure 1 5 is a flowchart illustrating the steps per- 
formed to create and display a complex image using 
the present invention. Images are created by con- 
structing the ACG on the display screen in step 1 510, 
editing the source node contents in step 1520, and 
executing the ACG in step 1530. After ACG execu- 
tion, the constructed and any intermediate images in- 
dicated in the ACG can be displayed, for example, on 
the display screen in step 1 540. Should any changes 
in the constructed image be desired after execution 
of the ACG, the ACG can be edited in step 1550, the 
source node contents can be edited, and then the 
ACG can be re-executed . While an entire ACG can 
be constructed before editing any of the source, 
nodes, an operator will preferably edit the source 
nodes as they are created. Additionally, the ACG and 
source nodes can be modified and remodified prior to 
execution of the ACG. 

Figure 16 illustrates the steps performed to con- 
struct an ACG on a display screen. As described ear- 
lier, the graph view window is activated in step 1511, 
by, for example, the user providing the appropriate 
signal (called up from the Graph Ops menu). The user 
then inputs a signal (e.g., by manipulating a mouse in 
the graph view window and inputting mouse clicks 
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where nodes are desired in the window) to indicate 
where a node is desired. The operator is then prompt- 
ed to specify the node type in step 1513. If the node 
type is a source node, the operator may choose to edit 
the contents of the source node's image fragment at 
this time. The nodes are interconnected in step 1515. 
Interconnecting nodes requires that the operator in- 
put signals which specify the two nodes to be inter- 
connected in step 1516. This can be done, for exam- 
ple, by positioning the cursor over one node, depress- 
ing a key on the keyboard, depressing the button on 
the mouse, and dragging the cursor to the desired 
node. When the cursor is released (completing the 
drag) by releasing the keyboard and cursor buttons, 
the two nodes are connected. This is displayed on the 
screen as a line interconnecting the two nodes. Addi- 
tionally, the underlying data structure takes the out- 
put of the child node and uses it as an input to the par- 
ent node. Prompts can be provided to indicate that 
the specified child is not of the proper type input for 
the parent. These prompts are issued based upon 
evaluations of the previously described protocols for 
each node type. It is understood that the specific user 
manipulations for creating nodes, specifying node 
types, and interconnecting nodes can vary based 25 
upon the software used to implement the present in- 
vention. Additionally, the operator can connect newly 
created nodes to existing nodes as they are created, 
and does not have to wait until all nodes In the ACG 
are formed before connecting any of the nodes. 30 

Figure 1 7 is a flow diagram of the process for exe- 
cuting the image formation process defined by an 
ACG. In step 1531. an operator inputs an "execute 
ACG" command. At this time, the structure walker 
can determine whether the ACG is well formed in step 35 
1 532. A well formed ACG is a DAG all of whose ports 
have at least one edge connected to them and all of 
whose edges obey the type restrictions of their ports. 
If the ACG Is not well formed, the user is prompted in 
step 1533 so that they can edit the ACG prior to re- 40 
execution. If the ACG is well formed, operation pro- 
ceeds to step 1534 where the nodes of the ACG are 
evaluated from the fringes (leafs) to the roots, as dis- 
cussed above. 

Figures 18A and 18B illustrate the processes for 45 
adding new nodes and deleting nodes, respectively, 
from an existing ACG. Referring to Figure 18A, when 
an operator provides a control signal for adding a new 
node in step 1551 , the operator is prompted to specify 
the node type in step 1552. If the node type is a 50 
source node, the operator may specify the contents 
of the source node at this time. Once a new node Is 
defined, the operator then interconnects that node to 
one or more existing nodes in the ACG in step 1554. 
This is performed by specifying the nodes for Inter- 55 
connection to the newly defined node in step 1555. It 
is understood that the process of interconnecting two 
nodes can be performed at any time in order to edit 



an ACG. That Is, an already existing node can be con- 
nected to another existing node by performing step 
1555. 

When it is desired to delete nodes, the operator 
5 provides a control signal in step 1557 to delete the 
specified node. This can be done, for example by pos- 
itioning a cursor over a node and Inputting an appro- 
priate signal (e.g., hold down a keyboard key and 
mouse click over the node) indicating that this node 
10 should be deleted. Once the node is deleted, the in- 
terconnections which existed between that node and 
other nodes must be deleted in step 1558. This can 
be performed automatically when nodes are deleted. 
Additionally, Interconnections (edges) between exist- 
15 Ing nodes can be deleted by providing the appropriate 
signals to delete a specified edge. 

Claims 

20 

1. An electronic image generator, comprising: 

means for generating at least one output 
image capable of being displayed, the output im- 
age being generated as a result of performing no, 
one or some transformation operations on one or 
more source images; 

means for constructing and displaying a 
graphical flow diagram of an Image formation 
process used to generate the output image, the 
graphical flow diagram including a plurality of In- 
terconnected nodes which represent the source 
images, any transformation operations, and the 
output Image: and 

means for controlling the graphical display 
constructing means to form and modify said 
graphical flow diagram, wherein the output image 
generation is responsive to the said graphical 
flow diagram so as to generate the or each output 
image. 

2. Theelectronicimagegeneratorofdaim 1, where- 
in the graphical flow diagram is acyclic. 

3. The electronic image generator of daim 1 or 2, 
wherein the graphical flow diagram corresponds 
to an image formation process operating on plu- 
ral source images in which each source image 
may be subjected to transformation operations, 
in which the output of transformation operations 
may be themselves subjected to transformation 
operations, and In which both source images and 
the outputs of transformation operations may be 
selectively used as the output Images. 

4. The electronic image generator of daim 3, where- 
in output images result directly or indirectly from 
all applications of transformation operations. 
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5. The electronic image generator of any preceding 
claim, wherein the control means is responsive to 
operator input commands. 

6. The electronic image generator of any proceed- 5 
ing claim, wherein the control means causes the 
means for generating a constructed image to dis- 
play the constructed image concurrently with the 
displayed graphical flow diagram. 

10 

7. The electronic image generator of any preceding 
claim, wherein the graphical flow diagram has 
nodes which are generated and displayed in a 
vertically-ordered manner, such that the nodes 
representing transformation operations appear is 
above the nodes representing source images or 
other transformation operations to which they 

are being applied, and the nodes representing 
output images appear above the nodes repre- 
senting source images or the transformation op- 20 
erations to which they are connected. 

8. The electronic image generator of any preceed- 
ing claim, wherein the control means permits the 
selection of source images and transformation 25 
operations from a library. 

9. The electronic image generator of claim 8, further 
comprising means for defining and entering addi- 
tional source images and transformation opera- 30 
tions into the library. 

10. The electronic image generator of any preceding 
claim, further comprising means for translating 
image data defining the source images or outputs 35 
of transformation operations into different graph- 
ical domains to permit the image data to be proc- 
essed by transformation operations operating in 
different graphical domains. 

4C 

1. The electronic image generator of claim 10, 
wherein the means for translating permits trans- 
lation of image data into a graphical domain suit- 
able for display. 

45 

2. The electronic image generator of claim lOorll, 
wherein the different graphical domains include 
at least two-dimensional rasters and two-dimen- 
sional vectors. 

50 

3. The electronic image generator of daim 12, 
wherein the transformation operations selective- 
ly operate in a single graphical domain or across 
different graphical domains so that each transfor- 
mation operation has input protocols and output 55 
protocols, and wherein the control means con- 
trols the means for constructing and displaying a 
graphical flow diagram so that the graphical flow 



diagram is constructed according to the input and 
output protocols. 

14. The electronic image generator of any preceding 
claim, wherein the means for generating one or 
more output images can use source images and 
the results of performing transformation opera- 
tions on source images as inputs to a plurality of 
further transformation operations, the means for 
constructing a graphical flow diagram displaying 
such plural input use by attaching the node asso- 
ciated with the source images and the results to 
the further transformation operations. 

1 5. The electronic image generator of any preceding 
claim, further comprising a display saeen, 
wherein the output images and the graphical flow 
diagram of an image formation sequence used to 
produce the output images are generated on the 
display screen. 

16. A method of generating a graphical representa- 
tion of an image formation process, comprising: 

selecting at least one image source used 
as a component of one or more final output im- 
ages; 

selecting no, one or more transformation 
operations to be performed on the image sourc- 
es to produce, at least indirectly, the final output 
images: and 

generating a graphical flow diagram in- 
cluding display icons representing all selected 
image sources, all selected image transformation 
operations, all selected final output images, and 
interconnections between the image sources, 
the image transformation operations, and the fi- 
nal output images which represent the image for- 
mation sequence which is performed to produce 
the final output images. 

17. A method of constructing a graphics image rep- 
resented by an output image display- icon from 
one or more image sources represented by image 
source display icons and no, one or more trans- 
formation operations represented by transforma- 
tion operation display icons, each transformation 
operation capable of being performed on at least 
one of the image sources, the method compris- 
ing: 

selecting at least one of the image sources 
as a component of a final output image; 

optionally selecting a transformation oper- 
ation to be performed on the image source to pro- 
duce, at least indirectly, the final output image; 

generating a graphical flow diagram of an 
image formation sequence required to produce 
the final output image by generating on a display 
screen the display icons associated with the se- 
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lected image sources, the selected transforma- 
tion operations, the final output image, and inter- 
connections between the image source display 
icons, the transformation operation display 
icons, and the output image display icon which s 
represent the image formation sequence which is 
performed to produce the final output image, and 
generating the final output image by exe- 
cuting the selected transformation operations on 
the selected image sources according to the to 
graphical flow diagram. 
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